Broker node EBS provisioned throughput settings #101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
angryhamsterx
force-pushed
the
broker_ebs_provisioned_throughput
branch
from
ea0fc1a
to
385f1f8
Compare
sbogdanovic
reviewed
Jan 30, 2024
| @@ -145,6 +145,15 @@ resource "aws_msk_cluster" "default" { | |||
| storage_info { | |||
| ebs_storage_info { | |||
| volume_size = var.broker_volume_size | |||
|
|
|||
| dynamic "provisioned_throughput" { | |||
There was a problem hiding this comment.
In case we want to explicitly disable provisioned throughput, to produce for example:
provisioned_throughput {
enabled = false
}the module with the change you suggested would try to set volume_throughput to the default value (250).
this would cover that case, and fix this bug:
dynamic "provisioned_throughput" {
for_each = var.broker_ebs_provisioned_throughput_enabled && var.broker_ebs_provisioned_throughput_state ? [1] : []
content {
enabled = true
volume_throughput = var.broker_ebs_provisioned_throughput_value
}
}
dynamic "provisioned_throughput" {
for_each = var.broker_ebs_provisioned_throughput_enabled && !var.broker_ebs_provisioned_throughput_state ? [1] : []
content {
enabled = false
}
}
There was a problem hiding this comment.
@angryhamsterx would you mind if we pull your PR changes into a seperate PR from us, with cloudposse'isms covered, and drive the PR forwards?
There was a problem hiding this comment.
This was referenced Feb 23, 2024
Merged
hans-d
added
wip
|
@angryhamsterx can you update the pr (see also the failing readme check) |
|
Thanks @angryhamsterx for creating this pull request! A maintainer will review your changes shortly. Please don't be discouraged if it takes a while. While you wait, make sure to review our contributor guidelines. Tip Need help or want to ask for a PR review to be expedited?Join us on Slack in the |
Nuru
requested changes
Jul 3, 2024
There was a problem hiding this comment.
Also, to fix #117 restore the
lifecycle {
ignore_changes = [
# Ignore changes to ebs_volume_size in favor of autoscaling policy
broker_node_group_info[0].storage_info[0].ebs_storage_info[0].volume_size,
]
}
We do NOT want to ignore changes to provisioned throughput, as that would make it impossible to update.
Comment on lines
+34
to
+58
| variable "broker_ebs_provisioned_throughput_enabled" { | ||
| type = bool | ||
| default = false | ||
| description = "Enable broker node EBS provisioned throughput for broker type kafka.m5.4xlarge or larger" | ||
| nullable = false | ||
| } | ||
|
|
||
| variable "broker_ebs_provisioned_throughput_state" { | ||
| type = bool | ||
| default = false | ||
| description = "Enable or disable provisioned throughput of the EBS volumes for the data drive on kafka broker nodes" | ||
| nullable = false | ||
| } | ||
|
|
||
| variable "broker_ebs_provisioned_throughput_value" { | ||
| type = number | ||
| default = 250 | ||
| description = "Provisioned throughput value of the EBS volumes for the data drive on kafka broker nodes in MiB per second" | ||
| validation { | ||
| condition = var.broker_ebs_provisioned_throughput_value >= 250 | ||
| error_message = "broker_ebs_provisioned_throughput_value must be minimum 250." | ||
| } | ||
| nullable = false | ||
| } | ||
|
|
There was a problem hiding this comment.
Suggested change
| variable "broker_ebs_provisioned_throughput_enabled" { | |
| type = bool | |
| default = false | |
| description = "Enable broker node EBS provisioned throughput for broker type kafka.m5.4xlarge or larger" | |
| nullable = false | |
| } | |
| variable "broker_ebs_provisioned_throughput_state" { | |
| type = bool | |
| default = false | |
| description = "Enable or disable provisioned throughput of the EBS volumes for the data drive on kafka broker nodes" | |
| nullable = false | |
| } | |
| variable "broker_ebs_provisioned_throughput_value" { | |
| type = number | |
| default = 250 | |
| description = "Provisioned throughput value of the EBS volumes for the data drive on kafka broker nodes in MiB per second" | |
| validation { | |
| condition = var.broker_ebs_provisioned_throughput_value >= 250 | |
| error_message = "broker_ebs_provisioned_throughput_value must be minimum 250." | |
| } | |
| nullable = false | |
| } | |
| variable "broker_ebs_provisioned_throughput" { | |
| type = object({ | |
| enabled = optional(bool, false) | |
| volume_throughput = optional(number) | |
| }) | |
| description = "Enable broker node EBS provisioned throughput for broker type kafka.m5.4xlarge or larger" | |
| default = {} | |
| nullable = false | |
| } | |
Comment on lines
+149
to
+156
| dynamic "provisioned_throughput" { | ||
| for_each = var.broker_ebs_provisioned_throughput_enabled ? [1] : [] | ||
|
|
||
| content { | ||
| enabled = var.broker_ebs_provisioned_throughput_state | ||
| volume_throughput = var.broker_ebs_provisioned_throughput_value | ||
| } | ||
| } |
There was a problem hiding this comment.
Suggested change
| dynamic "provisioned_throughput" { | |
| for_each = var.broker_ebs_provisioned_throughput_enabled ? [1] : [] | |
| content { | |
| enabled = var.broker_ebs_provisioned_throughput_state | |
| volume_throughput = var.broker_ebs_provisioned_throughput_value | |
| } | |
| } | |
| provisioned_throughput { | |
| enabled = var.broker_ebs_provisioned_throughput.enabled | |
| volume_throughput = var.broker_ebs_provisioned_throughput.volume_throughput | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
what
Ability to enable EBS volume provisioned throughput for broker nodes kafka.m5.4xlarge or larger.
You can add configuration block for this by setting
broker_ebs_provisioned_throughput_enabled, and then change enable/disable state bybroker_ebs_provisioned_throughput_state.Caveat:
Once
provisioned_throughputblock edded, it can not be removed, so there is variablebroker_ebs_provisioned_throughput_stateto change state of provisioned throughput.why
references